Image contrast enhancement in depth sensor

ABSTRACT

Embodiments related to the enhancement of contrast in an image pattern in a structured light depth sensor are disclosed. For example, one disclosed embodiment provides, in a structured light depth sensor system comprising a structured light depth sensor, a method comprising projecting a light pattern onto an object, detecting via an image sensor an image of the light pattern as reflected from the object, increasing a contrast of the light pattern relative to ambient light present in the image of the light pattern as reflected from the object to form a contrast-enhanced image of the light pattern as reflected from the object, and based upon a motion of the object as detected via the contrast-enhanced image of the light pattern, controlling an application that is providing output to a display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/472,921, titled IMAGE CONTRAST ENHANCEMENT IN DEPTH SENSOR and filedMay 27, 2009, the disclosure of which is hereby incorporated byreference in its entirety.

BACKGROUND

Image-based depth-sensors may be used in a variety of differentenvironments. For example, an image-based depth sensor may be used witha video game system to allow players to interact with the video gamesystem through the use of bodily gestures alone, without the use ofhand-held motion sensors or the like to detect the gestures.

Some image-based depth sensors utilize structured light to sense depthin an image. In such systems, a projector is used to illuminate a target(object) with a predefined light pattern. An image of this light patternas reflected by the target is then acquired via an image sensor, anddepth information is calculated from the distortion of the patternrelative to a known reference pattern in the image caused by the shapeof objects in the target. The performance of such image-based depthsensors may be dependent upon the contrast of the light pattern in theimage, which may be dependent upon the intensity and nature of ambientlight.

SUMMARY

Accordingly, various embodiments are described herein that are relatedto the enhancement of contrast in an image pattern in a structured lightdepth sensor. For example, one disclosed embodiment provides, in astructured light depth sensor system comprising a structured light depthsensor, a method comprising projecting a light pattern onto an object,detecting via an image sensor an image of the light pattern as reflectedfrom the object, increasing a contrast of the light pattern relative toambient light present in the image of the light pattern as reflectedfrom the object to form a contrast-enhanced image of the light patternas reflected from the object, and based upon a motion of the object asdetected via the contrast-enhanced image of the light pattern,controlling an application that is providing output to a display.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an embodiment of an example use environment for astructured light depth sensor.

FIG. 2 shows a block diagram of an example embodiment of a structuredlight depth sensor.

FIG. 3 shows a flow diagram depicting an embodiment of a method ofoperating a structured light depth sensor.

FIG. 4 shows a timing diagram that illustrates two example embodimentsof methods of modulating an intensity of a projected structured lightpattern.

FIG. 5 shows a block diagram of another example embodiment of astructured light depth sensor.

FIG. 6 shows a block diagram of another embodiment of a structured lightdepth sensor.

DETAILED DESCRIPTION

As mentioned above, the performance of a structured light depth sensormay be affected by the intensity and nature of ambient light presentwhen an image is captured for depth analysis. To avoid problems withambient light, a structured light depth sensor may include a narrowbandpass filter, matched to the wavelength of the laser, that limits thewavelengths of light that reach the depth image sensor. However, variousforms of ambient light may be present in wavelengths that can passthrough such a bandpass filter. For example, a structured light depthsensor with a bandpass filter configured to pass light in thenear-infrared spectrum may be affected by ambient light sources such asincandescent lights and sunlight, as such sources emit broadly acrossthe infrared spectrum. Depending upon the intensity of ambient lightfrom such sources in a desired use environment, the presence of suchambient light may make it difficult to detect the structured lightpattern in an acquired image.

Therefore, various embodiments are disclosed herein related to theenhancement of image contrast in a structured light depth sensor. Priorto discussing image contrast enhancement, an embodiment of an exampleuse environment is discussed with reference to FIG. 1. In particular,FIG. 1 shows a computer gaming system 12 that may be used to play avariety of different games, play one or more different media types,and/or control or manipulate non-game applications. FIG. 1 also shows adisplay 14 in the form of a television 16 that may be used to presentgame visuals to game players, such as game player 18. Furthermore, FIG.1 shows a capture device in the form of a depth sensor 20, which may beused to visually monitor one or more game players, such as game player18.

The depth sensor 20 may be used in combination with software on thegaming system 12 to track one or more targets, such as game player 18,in the field of view (“target”) of the depth sensor 20, by comparingimages of the targets taken at different times to detect motion. Gamingsystem 12 may then display a response to the motion on the television16. FIG. 1 shows a scenario in which game player 18 is tracked usingdepth sensor 20 so that the movements of game player 18 may beinterpreted by gaming system 12 as controls that can be used to affectthe game being executed by gaming system 12. In other words, game player18 may use his movements to control the game. The movements of gameplayer 18 may be interpreted as virtually any type of game control.

The example scenario illustrated in FIG. 1 shows game player 18 playinga boxing game that is being executed by gaming system 12. The gamingsystem uses television 16 to visually present a boxing opponent 22 togame player 18. Furthermore, the gaming system uses television 16 tovisually present a player avatar 24 that gaming player 18 controls withhis movements. In one example scenario, game player 18 can throw a punchin physical space as an instruction for player avatar 24 to throw apunch in game space. Gaming system 12 and depth sensor 20 can be used torecognize and analyze the punch of game player 18 in physical space sothat the punch can be interpreted as a game control that causes playeravatar 24 to throw a punch in game space. Likewise, other movements bygame player 18 may be interpreted as other controls, such as controls tobob, weave, shuffle, block, jab, or throw a variety of different powerpunches. Furthermore, some movements may be interpreted into controlsthat serve purposes other than controlling player avatar 24. Forexample, the player may use movements to end, pause, or save a game,select a level, view high scores, communicate with a friend, etc. Itwill be understood that the use environment of FIG. 1 is shown for thepurpose of example, and that a structured depth sensor may be used inany other suitable use environment.

FIG. 2 shows a block diagram of an example embodiment of a structuredlight depth sensor system 200. The structured light depth sensor system200 comprises a light pattern projector 202 configured to project apredetermined light pattern, and an image sensor 204 configured todetect an image of the light pattern as reflected by an object 206located in the field of view (FOV). The light pattern projector 202 andthe image sensor 204 are each in communication with a computing device208 configured to control the light pattern projector, to control andreceive data from the image sensor, and to analyze the data receivedfrom the image sensor to determine depth values for the locations in theFOV, e.g. at each pixel in the image. As such, the computing devicecomprises a processor 210 and memory 212 containing instructionsexecutable by the processor to perform various tasks related to thesefunctions, as well as any other suitable functions.

In some embodiments, the computing device 208 comprises an on-boardcontroller that is contained in a single housing with the light patternprojector 202 and the image sensor 204. In other embodiments, thecomputing device 208 may be housed separately from the light patternprojector 202 and the image sensor 204, e.g. as a desktop computer,laptop computer, server, etc. Likewise, in some embodiments, the lightpattern projector 202 and image sensor 204 may be contained within asingle housing, while in other embodiments, the light pattern projector202 and image sensor 204 may be separate components that are calibratedto one another based upon their relative positions.

Continuing with FIG. 2, in some embodiments, the structured light depthsensor system 200 may be configured to provide output to a display 214.For example, where the computing device 208 is a video game console, thecomputing device 208 may be configured to provide output to atelevision, monitor, or other display device to display feedback toplayer movements via the video game. It will be understood that theblock diagram shown in FIG. 2 is presented for the purpose of example,and that a structured light depth sensor system may have any othersuitable configuration than that shown.

As mentioned above, in some situations, ambient light may reduce acontrast of a structured light pattern in an image. This may make thedetermination of distance values from the structured light pattern inthe image more difficult. This problem may be particularly evident underconditions with high intensity ambient light from a broadband source,e.g. bright sunlight. Therefore, FIG. 3 shows a flow diagram depictingan embodiment of a method 300 of operating a structured light depthsensor system to aid in depth detection where ambient light may reduceimage contrast.

First, method 300 comprises, at 302, projecting a light pattern onto atarget. The light pattern may be created and projected in any suitablemanner. For example, in some embodiments, a laser projector is used tocreate a structured light pattern in the form of a speckle pattern.Likewise, a diffraction grating or other diffractive optical element maybe used in combination with a laser to create a diffraction pattern. Inyet other embodiments, a scanning laser projector may be used to createa pattern. Further, incoherent light sources also may be used to createa structured light pattern. For example, an incoherent light source maybe used in combination with a collimating lens and a light valve such asa liquid crystal display (LCD) panel, a digital light processing (DLP)chip, or the like, to create an image for projection onto a scene. Itwill be understood that these examples of methods to project astructured light pattern are presented for the purpose of example, andare not intended to be limiting in any manner.

While projecting the structured light pattern onto the target, method300 comprises, at 304, acquiring an image of the structured lightpattern as reflected from objects in the target, and at 306, increasinga contrast of the reflected structured light pattern relative to ambientlight in the image of the structured light pattern to form acontrast-enhanced image of the structured light pattern as reflectedfrom the target. Then, distance values for locations in the target aredetermined at 314, for example, as distance values for each pixel in theimage. Next, at 316, method 300 comprises comparing thecontrast-enhanced image to a previously acquired contrast-enhanced imageto detect motion of the target, and if motion of the target is detected,then displaying, at 318, a response to the motion on a display. Forexample, in the specific context of the video game system shown in FIG.1, a response may be displayed in the form of motion performed by anavatar displayed in the video game, and/or by performing a video gamecontrol function, such as pause/resume game play, turning on or offvideo game system power, etc.

The contrast of the structured light image may be increased in anysuitable manner. For example, in some embodiments and as indicated at308, a variance filter may be applied to the image acquired at 304 toform a variance image, and then the variance image may be used todetermine distance values. In other embodiments, as indicated at 310,the intensity of the projected structured light pattern may be modulatedat a rate substantially greater than a rate at which ambient lightintensities generally change, and a plurality of images may be acquiredat different projected light intensities. Then, a first image may besubtracted from a second image to correct for ambient light. In yetother embodiments, as indicated at 312, the structured image may beprojected via polarized light. This allows the use of a polarizationanalyzer or filter located between the target and the image sensor (forexample, at the image sensor) to reduce ambient light.

First referring to the application of a variance filter at 308, thevariance filter determines at each pixel in an image relative to one ormore nearby pixels. Thus, the variance image has higher pixel amplitudesin regions of higher gradient between pixels, and lower pixel amplitudesin regions of lower gradient between pixels. As such, the variance imagemay help to locate the borders of the structured light patterns in animage, as background regions between these borders may have values closeto zero in the variance image even in the presence of significantambient light intensities. The use of a variance filter to increaseimage contrast may offer the advantage that contrast may be increasedthrough software and/or firmware implementations alone, without anyhardware modifications.

Any suitable variance filter may be applied. For example, in someembodiments, an m×n matrix of pixels (where m and n are positivenon-zero integers, and at least one of m and n is greater than 1) isselected around each pixel in the image, and the statistical variance ofthe matrix is calculated. Performing this process over all pixels of theimage sensor yields the variance image. Examples of suitable m×nmatrices include, but are not limited to, 2×2 and 3×3 matrices. It willbe understood that other statistical methods may be used to calculate avariance image without departing from the scope of disclosure. It willbe understood that the terms “variance image”, “variance filter”, andthe like as used herein refer to any image and filter for creating suchan image that increase a contrast of an image based upon gradientsbetween pixels in an image.

Next referring to the modulation of light pattern intensity at 310,during ordinary use conditions, ambient light intensities generallychange at relatively low frequencies. For example, in exterior useenvironments, ambient light changes as passing clouds modify theintensity of sunlight, as the sun rises and sets, etc. Likewise, ininterior use environments, ambient light intensities change as lightsare turned on and off, etc. Therefore, under such use conditions, theambient light may have relatively constant values over time frames ofseconds, minutes, or even hours. Therefore, modulating the intensity ofstructured light image may allow images to be acquired that haveapproximately equal ambient light intensity levels but differentstructured light pattern intensity levels. Thus, a first image of thetarget at a first, higher light pattern intensity and to a second imageof the target at a second, lower light pattern intensity may besubtracted to substantially remove ambient light from the two images,thereby increasing the contrast of the structured light image in theresulting image.

The intensity of the structured light pattern may be varied in anysuitable manner. For example, in one embodiment, the intensity of thestructured light pattern is turned on and off at a frequency of ½ theimage acquisition and integration speed of the image sensor such thatevery other image taken by the image sensor is taken with the patternand the subsequent image with ambient light but without the pattern. Animage taken with the pattern may then be mathematically manipulated withan image taken without the pattern, for example, via subtraction, toreduce or remove ambient light from the image, thereby improvingcontrast of the structured light pattern. In other embodiments, thestructured light pattern may be turned either on or off for more two ormore sequential frames such that ambient-only images are acquired eithermore or less frequently than every other image, and/or may be partiallydimmed instead of turned off.

FIG. 4 shows a timing diagram 400 illustrating two example embodimentsof methods of modulating an intensity of a projected structured lightimage. First, line 402 illustrates the operation of the image sensor.The image integration periods are shown as peaks 404 and sensor readingperiods are shown as troughs 406. Next, line 408 shows a first examplemethod of modulating a structured light intensity. As depicted,projected light intensity is modulated at ½ the frequency of the imagesensor frame rate. In this manner, every two sequential image framesacquired by the image sensor comprises one structured light patternimage and one ambient-only image. Thus, sequential may be used for asubtraction operation to increase image contrast by removing ambientlight from the image. In other embodiments where non-sequential imageshave different structured light pattern intensities, non-sequentialimages may be subtracted to increase image contrast.

Line 410 depicts a second example method of modulating a structuredlight intensity. Whereas line 408 depicts the projected light remainingon for a full integration/read cycle, line 410 depicts the projectedlight remaining on for every other integration process, and then beingturned off for the corresponding read process and the nextintegration/read cycle. This may help to reduce power consumptionrelative to the method depicted via line 408. It will be appreciatedthat either of these methods, as well as other image intensitymodulation methods, may offer reduced power consumption relative to thecontinuous projection of a structured light image. The reduction intotal power also may help to increase the lifetime of the laser.

Some interior light may have a beat frequency of 60 Hz (or other) thatis due to the frequency of the AC power used to power the interior lightsources. Therefore, in such environments, the ambient light may vary atthis frequency. Thus, a depth sensor configured for use in suchenvironments may be configured to acquire images at a multiple orharmonic of 60 Hz (e.g. 30 Hz) so that the ambient light intensity isthe same for each image acquired. It will be understood that thespecific light modulation examples discussed herein are presented forthe purpose of example, and that any other suitable scheme may be usedto modulate the intensity of a projected structured light pattern tocorrect for ambient light without departing from the scope of thepresent disclosure.

Returning briefly to FIG. 3, as indicated at 312, in some embodiments, apolarization filter may be used to prevent some light reflected from thetarget from reaching the image sensor to help increase image contrast,either alone or in combination with other methods of increasing imagecontrast. FIGS. 5 and 6 show block diagrams of examples of the opticalcomponents of depth sensors that comprise polarization filters used incombination with image sensors. First regarding FIG. 5, a depth sensor500 is shown with projection optics 501 that comprise a laser projector502 used to produce a structured light pattern. The laser projector 502may be used in combination with an optional diffraction grating 504 tocreate the structured light pattern, or may create the structured lightpattern in any other suitable manner (e.g. via scanning, speckle, etc.).The laser projector 502 may emit light that has a relatively high degreeof polarization. Further, an optional polarization filter 506 may beused to increase a degree of polarization of light from the laser.

Polarized light from the laser that is reflected off of objects in thetarget may have a reduced polarization due to scattering duringreflection, but the reflected light may still have a partial degree oflinear polarization. On the other hand, ambient light sources such asthe sun and incandescent lighting produce unpolarized light. Therefore,image sensing optics 508 of depth sensor 500 may comprise an imagesensor 510, a bandpass filter 512, and a polarization filter 514. Thepolarization filter 514 allows light polarized along one orientation topass while that which is orthogonal will be blocked 510. Because thereflected structured light pattern has a predominant orientation, aslong as the polarization filter 514 is correctly aligned to thepredominant orientation of the reflected structure light pattern, thereduction in intensity of the reflected structured light pattern causedby the polarization filter 514 is less than the reduction of intensityof ambient light (50%) caused by the polarization filter 514. This mayhelp to increase the contrast in the image of the structured lightpattern. The bandpass filter 512 may be configured to pass a narrow bandof light that includes the wavelengths emitted by laser projector 502 tofurther reduce ambient light levels.

FIG. 6 shows a block diagram of another embodiment of a depth sensor 600configured to increase image contrast via a polarization filter. Theprojection optics of sensor 600 are shown at 601. Whereas depth sensor500 utilizes a coherent light source, depth sensor 600 utilizes anincoherent light source 602, such as one or more light-emitting diodes,configured to provide light to a light valve 604, such as an LCD panel,DLP chip, or the like, for projection of a structured light pattern.Where an LCD panel is used to project the structured light pattern, theLCD panel itself acts as a first polarization filter to producepolarized light. On the other hand, where the light valve 604 does notproduce polarized light (e.g. a DLP light valve), the projection optics601 also may comprise a separate first polarization filter 606 so thatthe structured light pattern is projected with polarized light.

As mentioned above, the structured light pattern reflected from objectin the target may retain a partial degree of linear orientation.Therefore, the image sensing optics 608 of depth sensor 600 comprise animage sensor 610, and a second polarization filter 614 configured topass light of the same orientation as the predominant orientation of thereflected structured light pattern. In this manner, the intensity ofambient light is reduced by a greater amount (50%) than the intensity ofthe reflected structured light pattern. Further, the image sensingoptics 608 of the depth sensor also may include a band pass filter 612configured to pass light of the wavelength or wavelengths emitted byincoherent light source 602.

It will be understood that the configurations and/or approachesdescribed herein for increasing image contrast in a structured lightdepth sensor are presented for the purpose of example and not intendedto be limiting, because numerous variations are possible. The specificroutines or methods described herein may represent one or more of anynumber of processing strategies. As such, various acts illustrated maybe performed in the sequence illustrated, in other sequences, inparallel, or in some cases omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnon-obvious combinations and subcombinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof

1. In a structured light depth sensor system comprising a structuredlight depth sensor, a method comprising: projecting a light pattern ontoan object; detecting via an image sensor an image of the light patternas reflected from the object; increasing a contrast of the light patternrelative to ambient light present in the image of the light pattern asreflected from the object to form a contrast-enhanced image of the lightpattern as reflected from the object; and based upon a motion of theobject as detected via the contrast-enhanced image of the light pattern,controlling an application that is providing output to a display.
 2. Themethod of claim 1, wherein projecting the light pattern comprisesmodulating an intensity of the light pattern as a function of time, andwherein increasing a contrast of the light pattern comprises subtractinga first image of the light pattern as reflected from the object at afirst light pattern intensity with a second image of the light patternas reflected from the object at a second light pattern intensity.
 3. Themethod of claim 2, wherein modulating an intensity of the light patterncomprises turning the light pattern on and off.
 4. The method of claim1, wherein controlling the application comprises displaying a motion ofan avatar on the display.
 5. The method of claim 1, wherein thestructured light depth sensor system comprises a computer gaming system.6. The method of claim 1, wherein increasing a contrast of the lightpattern comprises calculating a variance image from the image of thelight pattern as reflected from the object.
 7. The method of claim 6,wherein determining the variance image comprises, for each pixel,determining a variance value from an m×n array of nearby pixels, whereinat least one of m and n has a value of at least
 2. 8. The method ofclaim 6, wherein projecting the light pattern onto the object comprisesprojecting the light pattern via polarized light, and wherein increasingthe contrast of the light pattern comprises filtering the light patternreflected from the object via a polarization filter positioned opticallybetween the object and the image sensor.
 9. The method of claim 1,wherein projecting the light pattern onto the object comprisesprojecting the light pattern via polarized light, and wherein increasingthe contrast of the light pattern comprises filtering the light patternreflected from the object scene via a polarization filter positionedoptically between the object and the image sensor.
 10. The method ofclaim 9, wherein projecting the light pattern via polarized lightcomprises projecting the light pattern via a laser.
 11. The method ofclaim 9, wherein projecting the light pattern via polarized lightcomprises producing the light pattern via an incoherent light sourcecombined with another polarization filter.
 12. An optical depth-sensingsystem, comprising: an image source configured to project a lightpattern onto an object; an image sensor configured to capture an imageof light reflected from the object; a processor in operativecommunication with the image source, the display, and the image sensor;and memory comprising instructions stored thereon that are executable bythe processor to: control projection of the light pattern onto theobject; receive from the image sensor an image of the light pattern asreflected from the object; increase a contrast of the light patternrelative to ambient light in the image of the light pattern as reflectedfrom the object to form a contrast-enhanced image of the light patternas reflected from the object; detect motion of the object via thecontrast-enhanced image; and in response to the motion, control anapplication that is providing an output to a display.
 13. The apparatusof claim 12, wherein the instructions are executable to controlprojection of the light pattern onto the object by modulating anintensity of the light pattern as a function of time, and wherein theinstructions are executable to increase the contrast of the lightpattern by comparing a first image of the object at a first, higherlight pattern intensity to a second image of the object at a second,lower light pattern intensity.
 14. The apparatus of claim 12, whereinthe instructions are executable to control the application bycontrolling a display of a motion of an avatar on the display.
 15. Theapparatus of claim 12, wherein the optical depth sensing systemcomprises a computer gaming system.
 16. A computer gaming system,comprising: an image source configured to project a light pattern onto atarget; an image sensor configured to capture an image of the lightpattern reflected from the target; a processor in operativecommunication with the image source and the image sensor; and memorycomprising instructions stored thereon that are executable by theprocessor to: control projection of the light pattern onto the target;receive from the image sensor an image of the light pattern as reflectedfrom the target; increase a contrast of the light pattern as reflectedfrom the target to form a contrast-enhanced image of the light patternas reflected from the target; compare the contrast-enhanced image to apreviously acquired contrast-enhanced image to detect motion of thetarget; and in response to detecting motion of the target, displaymotion of an avatar on a display.
 17. The computer gaming system ofclaim 16, wherein the image source comprises a laser.
 18. The computergaming system of claim 16, further comprising a first polarizationfilter located between the target and the image sensor, and whereinimage source comprises an incoherent light source and a secondpolarization filter configured to polarize light from the incoherentlight source.
 19. The computer gaming system of claim 16, wherein theinstructions are further executable to modulate an intensity of thelight pattern as a function of time, and to increase the contrast of thelight pattern by comparing a first image of the target at a first,higher light pattern intensity to a second image of the target at asecond, lower light pattern intensity.
 20. The computer gaming system ofclaim 16, wherein the instructions are further executable to construct avariance image from the image of the light pattern detected via theimage sensor, and to determine a distance value for each of one or morelocations in target via the variance image.